Learning Activation Functions in Deep (Spline) Neural Networks
نویسندگان
چکیده
منابع مشابه
Learning Activation Functions to Improve Deep Neural Networks
Artificial neural networks typically have a fixed, non-linear activation function at each neuron. We have designed a novel form of piecewise linear activation function that is learned independently for each neuron using gradient descent. With this adaptive activation function, we are able to improve upon deep neural network architectures composed of static rectified linear units, achieving stat...
متن کاملDeep Neural Networks with Multistate Activation Functions
We propose multistate activation functions (MSAFs) for deep neural networks (DNNs). These MSAFs are new kinds of activation functions which are capable of representing more than two states, including the N-order MSAFs and the symmetrical MSAF. DNNs with these MSAFs can be trained via conventional Stochastic Gradient Descent (SGD) as well as mean-normalised SGD. We also discuss how these MSAFs p...
متن کاملArtificial Neural Networks with Adaptive Multidimensional Spline Activation Functions
This work concerns a new kind of neural structure that involves a multidimensional adaptive activation function. The proposed architecture, based on multi-dimensional cubic spline, allows to collect information from the previous network layer in aggregate form. In other words the number of network connections (structural complexity) can be very low respect to the problem complexity. This fact, ...
متن کاملNonparametrically Learning Activation Functions in Deep Neural Nets
We provide a principled framework for nonparametrically learning activation functions in deep neural networks. Currently, state-of-the-art deep networks treat choice of activation function as a hyper-parameter before training. By allowing activation functions to be estimated as part of the training procedure, we expand the class of functions that each node in the network can learn. We also prov...
متن کاملReinforcement Learning in Neural Networks: A Survey
In recent years, researches on reinforcement learning (RL) have focused on bridging the gap between adaptive optimal control and bio-inspired learning techniques. Neural network reinforcement learning (NNRL) is among the most popular algorithms in the RL framework. The advantage of using neural networks enables the RL to search for optimal policies more efficiently in several real-life applicat...
متن کاملذخیره در منابع من
با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید
ژورنال
عنوان ژورنال: IEEE Open Journal of Signal Processing
سال: 2020
ISSN: 2644-1322
DOI: 10.1109/ojsp.2020.3039379